Image encoding/decoding apparatus, image processing system, image encoding/decoding method and training method

ABSTRACT

Provided are an image encoding apparatus, an image decoding apparatus, an image processing system including the image encoding/decoding apparatus, a training method for the image processing system and a display apparatus. The image encoding apparatus includes: a first image input terminal (403) configured to provide a first image (UL) and a plurality of second image input terminals (404, 405, 406) configured to provide a plurality of second images (UR, BR, BL); a first convolutional neural network module (47), configured to update the plurality of second images to obtain corresponding update features; an image superposing module (408), configured to superpose the update features with the first image according to a superposing weight to generate superposed images and output the superposed images through an output interface (411); a second convolutional neural network module (409), configured to generate a plurality of prediction images according to the superposed images; and an image difference acquisition module (410), configured to determine difference features between each second image of the plurality of second images and a corresponding prediction image and output the difference features through the output interface.

CROSS-REFERENCE TO RELATED APPLICATION

This application is the National Stage of PCT/CN2017/090260 filed onJun. 27, 2017, which claims priority under 35 U.S.C. § 119 of a Chinesepatent application No. 201610885633.4 filed on Oct. 11, 2016, which ishereby incorporated by reference in its entirety as a part of thepresent application.

TECHNICAL FIELD

The present disclosure relates to an image encoding apparatus, an imagedecoding apparatus, an image processing system comprising the imageencoding/decoding apparatus, a training method for the image processingsystem and a display apparatus.

BACKGROUND

In recent years, the quality of digital images and videos is improvedquickly with standards such as BT.2020 established. The improvement inimage quality is tied to a significant increase in data size. At thesame time, the most popular source of media comes from the Internet. Itis expected that even though bandwidth is increasing steadily, adramatic increase in media data traffic is hard to be satisfied.Therefore, it needs to seek for better solutions for media datacompression to satisfy the requirement for high quality media data underthe existing traffic bandwidth.

SUMMARY

According to one aspect of the present disclosure, there is provided inan embodiment of the present disclosure an image encoding apparatus,comprising: a first image input terminal, configured to provide a firstimage; a plurality of second image input terminals, configured toprovide a plurality of second images; a first convolutional neuralnetwork circuit connected with the plurality of second image inputterminals and configured to update features of each second image of theplurality of second images to obtain corresponding update features; animage superposing circuit, connected with the first image input terminaland the first convolutional neural network circuit and configured tosuperpose the update feature of each second image of the plurality ofsecond images with the first image to generate superposed images andoutput the superposed images; a prediction circuit, connected with theimage superposing circuit and configured to generate a plurality ofprediction images according to the superposed images; an imagedifference acquisition circuit, connected with the plurality of secondimage input terminals and the prediction circuit and configured todetermine difference features between each second image of the pluralityof second images and the corresponding prediction image and output thedifference features; and an output interface, configured to output thesuperposed images and the difference features.

Optionally, in the image encoding apparatus, the prediction circuit is asecond convolutional neural network circuit.

According to another aspect of the present disclosure, there is providedan image encoding apparatus, comprising: a first image input terminal,configured to acquire a first image; a plurality of second image inputterminals, configured to acquire a plurality of second images; a featurecircuit, connected with the plurality of second image input terminalsand configured to update features of each second image of the pluralityof second images to obtain corresponding update features; an imagesuperposing circuit, connected with the first image input terminal andthe feature circuit and configured to superpose the update features ofeach second image of the plurality of second images with the first imageto generate superposed images and output the superposed images; a secondconvolutional neural network circuit, connected with the imagesuperposing circuit and configured to generate a plurality of predictionimages according to each of the superposed images; an image differenceacquisition circuit, connected with the plurality of second image inputterminals and the prediction circuit and configured to determinedifference features between each second image of the plurality of secondimages and a corresponding prediction image and output the differencefeatures; and an output interface, configured to output the superposedimages and the difference features.

Optionally, the image encoding apparatus further comprises: a demuxer,connected with the first image input terminal and the plurality ofsecond image input terminals and configured to split an input originalimage to obtain the first image and the plurality of second images.

Optionally, in the image encoding apparatus described above, the imagesuperposing circuit superposes the update features of each second imageof the plurality of second images with the first image according to asuperposing weight.

In the image encoding apparatus according to an embodiment of thepresent disclosure, the image superposing circuit is configured tomultiply the first image by a first weight parameter to obtain a firstproduct, multiply the update features by a second weight parameter toobtain a second product, and superpose the first product and the secondproduct to generate superposed images; wherein the first weightparameter is greater than 0, and a sum of the first weight parameter andthe second weight parameter is 1.

The image encoding apparatus according to an embodiment of the presentdisclosure further comprises: a demuxer, connected with the first imageinput terminal and the plurality of second image input terminals andconfigured to split an input original image to obtain the first imageand the plurality of second images.

Optionally, in the image encoding apparatus according to an embodimentof the present disclosure, the demuxer is configured to split theoriginal image into 2n images, a number of the first image is 1, anumber of the second images is 2n−1, and n is an integer greater than 0.

According to another aspect of the present disclosure, there is providedin an embodiment of the present disclosure an image decoding apparatus,comprising: a superposed image input terminal, configured to receivesuperposed images; a difference feature input terminal, configured toreceive difference features; a prediction circuit, connected with thesuperposed image input terminal and configured to generate a pluralityof prediction images according to the superposed images; a de-differencecircuit, connected with the difference feature input terminal and theprediction circuit and configured to generate a plurality of secondimages according to the plurality of prediction images and thedifference features and output the plurality of second images; a fourthconvolutional neural network circuit, connected with the de-differencecircuit and configured to update each second image of the plurality ofsecond images to obtain a corresponding update feature; and an imagede-superposing circuit, connected with the superposed image inputterminal and the fourth convolutional neural network circuit andconfigured to perform de-superposing on the superposed images accordingto the update features to obtain a first image, and output the firstimage; an output terminal, configured to output the plurality of secondimages and the first image.

Optionally, in the image decoding apparatus, the prediction circuit is athird convolutional neural network circuit.

According to another aspect of the present disclosure, there is providedan image decoding apparatus, comprising: a superposed image inputterminal, configured to receive superposed images; a difference featureinput terminal, configured to receive difference features; a thirdconvolutional neural network circuit, connected with the superposedimage input terminal and configured to generate a plurality ofprediction images according to the superposed images; a de-differencecircuit, connected with the difference feature input terminal and thethird convolutional neural network and configured to generate aplurality of second images according to each prediction image of theplurality of prediction images and the difference features and outputthe plurality of second images; a feature circuit, connected with thede-difference circuit and configured to update each second image of theplurality of second images to obtain a corresponding update feature; andan image de-superposing circuit, connected with the superposed imageinput terminal and the feature circuit and configured to performde-superposing on the superposed images according to the update featuresto obtain a first image, and output the first image; and an outputterminal, configured to output the plurality of second images and thefirst image.

Optionally, the image decoding apparatus further comprises: a muxer,connected with the output terminal and configured to split joint thefirst image and the plurality of second images to obtain a decoded imageand output the decoded image through an output interface.

Optionally, in the above-mentioned image decoding apparatus, the imagede-superposing circuit is configured to perform de-superposing on thesuperposed images according to the update features and their superposingweights.

In the image decoding apparatus according to an embodiment of thepresent disclosure, the image de-superposing circuit is configured tomultiply the update features by a second weight parameter to obtain asecond product, remove the second product from the superposed images toobtain a first product, and divide the first product by a first weightparameter to obtain the first image; wherein the first weight parameteris greater than 0, and a sum of the first weight parameter and thesecond weight parameter is 1.

According to another aspect of the present disclosure, there is providedin an embodiment of the present disclosure an image processing system,comprising: an image encoding apparatus, comprising: a first image inputterminal, configured to acquire a first image; a plurality of secondimage input terminals, configured to acquire a plurality of secondimages; a first convolutional neural network circuit, connected with theplurality of second image input terminals and configured to updatefeatures of each second image of the plurality of second images toobtain corresponding update features; an image superposing circuit,connected with the first image input terminal and the firstconvolutional neural network circuit and configured to superpose theupdate features of each second image of the plurality of second imageswith the first image to generate superposed images and output thesuperposed images; a first prediction circuit, connected with the imagesuperposing circuit and configured to generate a plurality of predictionimages according to each of the superposed images; an image differenceacquisition circuit, connected with the plurality of second image inputterminals and the prediction circuit and configured to determinedifference features between each of the plurality of second images and acorresponding prediction image and output the difference features; andan output interface, configured to output the superposed images and thedifference features; an image decoding apparatus, comprising: asuperposed image input terminal, configured to receive the superposedimages; a difference feature input terminal, configured to receive thedifference features; a second prediction circuit, connected with thesuperposed image input terminal and configured to generate a pluralityof prediction images according to the superposed images; a de-differencecircuit, connected with the difference feature input terminal and theprediction circuit and configured to generate a plurality of fourthimages according to each prediction image of the plurality of predictionimages and the difference features and output the plurality of fourthimages; a fourth convolutional neural network circuit, connected withthe de-difference circuit and configured to update the plurality offourth images to obtain corresponding update features; and an imagede-superposing circuit, connected with the superposed image inputterminal and the fourth convolutional neural network circuit andconfigured to perform de-superposing on the superposed images accordingto the update features to obtain a third image, and output the thirdimage; and an output terminal, configured to output the plurality offourth images and the third image.

Optionally, in the image processing system described above, the firstprediction circuit is a second convolutional neural network circuit, andthe second prediction circuit is a third convolutional neural networkcircuit.

According to another aspect of the present disclosure, there is providedan image processing system, comprising: an image encoding apparatus,comprising: a first image input terminal, configured to acquire a firstimage; a plurality of second image input terminals, configured toacquire a plurality of second images; a first feature circuit, connectedwith the plurality of second image input terminals and configured toupdate features of each second image of the plurality of second imagesto obtain corresponding update features; an image superposing circuit,connected with the first image input terminal and the first featurecircuit and configured to superpose the update feature of each secondimage of the plurality of second images with the first image to generatesuperposed images and output the superposed images; a secondconvolutional neural network circuit, connected with the imagesuperposing circuit and configured to generate a plurality of predictionimages according to each of the superposed images; an image differenceacquisition circuit, connected with the plurality of second image inputterminals and the second convolutional neural network circuit andconfigured to determine difference features between each second image ofthe plurality of second images and a corresponding prediction image andoutput the difference features; and an output interface, configured tooutput the superposed images and the difference features; an imagedecoding apparatus, comprising: a superposed image input terminal,configured to receive the superposed images; a difference feature inputterminal, configured to receive the difference features; a thirdconvolutional neural network circuit, connected with the superposedimage input terminal and configured to generate a plurality ofprediction images according to the superposed images; a de-differencecircuit, connected with the difference feature input terminal and thethird convolutional neural network circuit and configured to generate aplurality of fourth images according to each prediction image of theplurality of prediction images and the difference features and outputthe plurality of fourth images; a second feature circuit, connected withthe de-difference circuit and configured to update the plurality offourth images to obtain corresponding update features; and an imagede-superposing circuit, connected with the superposed image inputterminal and the second feature circuit and configured to performde-superposing on the superposed images according to the update featuresto obtain a third image, and output the third image; and an outputterminal, configured to output the plurality of fourth images and thethird image.

Optionally, in the image processing system described above, the firstfeature circuit is a first convolutional neural network circuit, and thesecond feature circuit is a fourth convolutional neural network circuit.

Optionally, the image processing system further comprises a quantizationapparatus, connected with the image encoding apparatus and configured toreceive the superposed images and the difference features output fromthe output interface and perform quantization process and inversequantization process on the superposd images and the differencefeatures, to generate quantization superposed images and quantizationdifference features; and the image decoding apparatus, configured tooutput the quantization superposed image and the quantization differencefeatures to a superposed image input terminal and a difference featureinput terminal of the image decoding apparatus.

In the image processing system according to an embodiment of the presentdisclosure, the quantization apparatus is configured to perform thequantization process on the superposed images and the differencefeatures by utilizing an uniform scalar quantization USQ function,

${{USQ}(x)} = {{{sign}(x)}\left\lfloor \frac{x}{\delta} \right\rfloor}$

where

${{sign}(x)} = \left\{ {\left. \begin{matrix}1 & {x > 0} \\0 & {x = 0} \\{- 1} & {x < 0}\end{matrix} \right|,\left\lfloor \frac{x}{\delta} \right\rfloor} \right.$

is a smallest integer smaller than x, and δ is a quantization parameter.

In the image processing system according to an embodiment of the presentdisclosure, the quantization apparatus is configured to perform thequantization process on an output q of the uniform scalar quantizationUSQ function by utilizing an inverse uniform scalar quantization InvUSQfunction to generate the quantization superposed images and thequantization difference features,

where InvUSQ(q)=sign(q)(|q|+0.5)δ.

According to another aspect of the present disclosure, there is providedan image encoding method, comprising steps of:

acquiring a first image and a plurality of second images;

updating features of each second image of the plurality of second imagesto obtain corresponding update features;

superposing the first image and the update features of each second imageof the plurality of second images to generate superposed images;

generating a plurality of prediction images according to the superposedimages;

determining difference features between each second image of the secondimages and a corresponding prediction image;

outputting the superposed images and the difference features;

wherein the updating and/or predicting adopts a convolutional neuralnetwork.

Optionally, the image encoding method further comprises steps of:

splitting an input original image into the first image and the pluralityof second images.

According to another aspect of the present disclosure, there is providedan image decoding method, comprising steps of: receiving superposedimages and difference features; generating a plurality of predictionimages according to the superposed images; generating a plurality ofsecond images according to each prediction image of the plurality ofprediction images and the difference features; updating each secondimage of the plurality of second images to obtain corresponding updatefeatures; performing de-superposing on the superposed images accordingto the update features to obtain a first image; outputting the pluralityof second images and the first image; wherein the updating and/orpredicting adopts a convolutional neural network.

Optionally, the image decoding method further comprises a step of:joining the first image and the plurality of second images to obtain adecoded image.

According to another aspect of the present disclosure, there is providedin an embodiment of the present disclosure a training method for animage processing system, comprising: selecting a fixed quantizationparameter; inputting a training image to the image processing system,adjusting weight values of respective filter circuits at respectiveconvolutional layers in convolutional neural network circuits, andrunning a limited number of iterations to optimize a target function;and reducing the quantization parameter by a predetermined value, andrepeating a training step of optimizing the target function if thequantization parameter is not smaller than a predetermined threshold;otherwise, the training method ends up.

In the training method according to an embodiment of the presentdisclosure, the target function is:

θ=arg_(θ)min_(X)MSE(X,OUT_(θ)(X,δ))

where X represents an input training image, OUT represents an outputimage, and MSE is a mean square error function between the inputtraining image and the output image.

According to another aspect of the present application, there isprovided in an embodiment of the present disclosure a display apparatus,comprising the image encoding apparatus, the image decoding apparatusand/or the image processing system described above.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other purposes, features and advantages of the presentdisclosure would become more evident by describing embodiments of thepresent disclosure in more details in combination with accompanyingdrawings. In the drawings, same reference marks generally represent samemeans or step.

FIG. 1 is a schematic diagram illustrating a convolutional neuralnetwork for image processing;

FIG. 2 is a schematic diagram illustrating a wavelet transform for amulti-resolution image transform;

FIG. 3 is a schematic diagram of structure of an image processing systemthat utilizes a convolutional neural network to realize a wavelettransform;

FIG. 4 is a schematic diagram illustrating structure of an imageencoding apparatus according to a first embodiment of the presentdisclosure;

FIG. 5 is a schematic diagram illustrating structure of an imagedecoding apparatus according to a first embodiment of the presentdisclosure;

FIG. 6 is a schematic diagram illustrating structure of an imageprocessing system according to a second embodiment of the presentdisclosure; and

FIG. 7 is a flowchart diagram illustrating a training method accordingto a third embodiment of the present disclosure.

DETAILED DESCRIPTION

In order to make principles, technical solutions and advantages of thepresent disclosure more evident, exemplary embodiments according to thepresent disclosure will be described below in detail by referring to theaccompanying drawings. Obviously, the embodiments described below arejust a part of embodiments of the present disclosure, rather than allthe embodiments of the present disclosure. It shall be understood thatthe present disclosure is not limited to the exemplary embodimentsdescribed herein. Based on the embodiments of the present disclosuredescribed in the present disclosure, all the other embodiments obtainedby those skilled in the art without paying any inventive labor shallfall into the protection scope of the present disclosure.

Before detailed description of an image encoding apparatus, an imagedecoding apparatus and an image processing system according toembodiments of the present disclosure, basic concepts of a convolutionalneural network being used for image encoding/decoding process aredescribed by referring to the accompanying drawings.

FIG. 1 illustrates a schematic diagram of a convolutional neural networkused for image processing. The convolutional neural network used forimage processing uses images as inputs/outputs and replaces scalarweights by filters (convolutions). FIG. 1 shows a convolutional neuralnetwork having a simple structure with 3 layers. As shown in FIG. 1, 4input images are input at an input layer 101, there are 3 units at ahidden layer 102 in the center to output 3 output images, and there are2 units at an output layer 103 to output 2 output images. Each box withweights w_(ij) ^(k) at the input layer 101 corresponds to a filter,where k is a label indicating an input layer number, and i and j arelabels indicating input and output units, respectively. The biases b_(i)^(k) are scalars added to the outputs of convolutions. The result ofadding several convolutions and bias then passes through an activationbox, that typically corresponds to a rectifying linear unit (ReLU), orsigmoid function, or hyperbolic tangent, etc. In the image processingsystem using the convolutional neural network, the respective filtersand biases are fixed during operation of the system. The filters andbiases are obtained by a process of training using a set of input/outputexample images in advance and adjusted to fit some optimizationcriterion that depends on applications.

FIG. 2 is a schematic diagram illustrating a wavelet transform for amulti-resolution image transform. Wavelet Transform is amulti-resolution image transform for image encoding/decoding process.Applications of wavelet transform include transform coding in JPEG 2000standard. In the image encoding (compression) process, the wavelettransform is used to represent an original high-resolution image interms of smaller low-resolution images (for example, a part of images ofthe original images). In the image decoding (decompression) process, aninverse wavelet transform is used to recover and obtain the originalimage by utilizing a low-resolution image and the difference featuresrequired for recovering the original image.

FIG. 2 schematically shows a 3-level wavelet transform and inversetransform. As shown in FIG. 2, one of the smaller low-resolution imagesis a downscale version A of the original image, and the otherlow-resolution images represent the missing details (Dh, Dv and Dd)needed to recover the original image.

FIG. 3 is a schematic diagram of structure of an image processing systemthat utilizes a convolutional neural network to realize a wavelettransform. Lifting Scheme is an efficient implementation of the wavelettransform and a flexible tool for constructing wavelets. A standardstructure for 1D data is shown illustratively in FIG. 3. The left sidein FIG. 3 corresponds to an encoder 31. A demuxer 302 in the encoder 31transforms an input original image 301 into a low-resolution image A anddetail D. The encoder 31 further uses a prediction filter p and anupdate filter u. For a compression application, it is desired that D≈0,so that most information is contained in A. The right side in FIG. 3corresponds to a decoder 32. The parameters of the decoder 32 areexactly the same filters p and u from the encoder 31, but the filters pand u are arranged inversely. Since the encoder 31 and the decoder 32are corresponding strictly, this arrangement ensures that a decodedimage 304 joined and obtained by the muxer 303 of the decoder is exactlythe same as the original image 301. In addition, the structure shown inFIG. 3 is not limited, and can be configured alternatively in thedecoder according to a sequence of an update filter u and a predictionfilter p.

An image encoding apparatus, an image decoding apparatus and an imageprocessing system comprising the image encoding/decoding apparatusaccording to embodiments of the present disclosure will be described indetail by referring to the accompanying drawings.

FIG. 4 is a schematic diagram illustrating structure of an imageencoding apparatus according to a first embodiment of the presentdisclosure.

As shown in FIG. 4, an image encoding apparatus 40 according to thefirst embodiment of the present disclosure comprises:

a demuxer 402, configured to split an input original image to obtain afirst image UL and a plurality of second images UR, BR, BL.

A first image input terminal 403 is configured to receive the firstimage UL from the demuxer 402. A plurality of second image inputterminals 404, 405, 406 is configured to receive the plurality of secondimages UR, BR, BL from the demuxer 402, respectively.

A first convolutional neural network 407 is connected with the pluralityof second image input terminals 404, 405, 406, and configured to updatethe plurality of second images UR, BR, BL to obtain corresponding updatefeatures.

In the first embodiment of the present disclosure, the firstconvolutional neural network circuit 407 can be an update filterdescribed by referring to FIG. 3.

An image superposing circuit 408 is connected with the first image inputterminal 403, the first convolutional neural network circuit 407 and anoutput interface 411 and configured to superpose the update features Uand the first image UL according to a superposing weight to generatesuperposed image A, and output the superposed image A through the outputinterface 411.

In an embodiment of the present disclosure, the image superposingcircuit 408 is configured to multiply the first image UL by a firstweight parameter a to obtain a first product, multiply the updatefeatures U by a second weight parameter b to obtain a second product,and superpose the first product and the second product to generate thesuperposed image A. The first weight parameter a is greater than 0, anda sum of the first weight parameter a and the second weight parameter bis 1. That is, in the image superposing circuit 408:

A=aUL+bU  Expression 1

a+b=1 and a>0  Expression 2

A second convolutional neural network circuit 409 is connected with theimage superposing circuit 408 and configured to generate a plurality ofprediction images according to the superposed image A.

An image difference acquisition circuit 410 is connected with theplurality of second image input terminals 404, 405, 406, the secondconvolutional neural network circuit 409 and the output interface 411and configured to determine difference features D_(n), D_(d) and D_(v)of each second image of the plurality of second images UR, BR, BL andcorresponding prediction images, and output the difference featuresD_(n), D_(d) and D_(v) through the output interface 411.

In the first embodiment of the present disclosure, a compression circuitof the image encoding apparatus 40 is composed of the firstconvolutional neural network circuit 407, the image superposing circuit408, the second convolutional neural network circuit 409 and the imagedifference acquisition circuit 410. The compression circuit performsimage compression based on lifting scheme of the wavelet transform onthe first image UL and the plurality of second images UR, BR, BL inputfrom the demuxer 402.

FIG. 5 is a schematic diagram illustrating structure of the imagedecoding apparatus according to the first embodiment of the presentdisclosure. The image decoding apparatus shown in FIG. 5 can be used todecode output images of the image encoding apparatus shown in FIG. 4.

As shown in FIG. 5, an image decoding apparatus 50 according to thefirst embodiment of the present disclosure comprises:

a superposed image input terminal 507, configured to receive thesuperposed image A. Difference feature input terminals 504, 505 and 506are configured to receive the difference features D_(n), D_(d) and D_(v)respectively. The superposed image A can be image data that comes fromthe image superposing circuit 408 and output from the output interface411 of the image encoding apparatus 40 shown in FIG. 4. The differencefeatures D_(n), D_(d) and D_(v) can be image data that comes from theimage difference acquisition circuit 410 and output from the outputinterface 411 of the image encoding apparatus 40 shown in FIG. 4.

A third convolutional neural network circuit 507 is connected with thesuperposed image input terminal 507 and configured to generate aplurality of prediction images according to the superposed image A.

A de-difference circuit 508 is connected with the difference featureinput terminals 504, 505 and 506, the third convolutional neural networkcircuit 507 and output terminals 512, 523 and 514, and configured togenerate a plurality of second images UR, BR, BL according to theplurality of prediction images and the difference features D_(n), D_(a)and D_(v), and output the plurality of second images through the outputterminals 512, 513 and 514.

A fourth convolutional neural network circuit 509 is connected with thede-difference circuit 508 and configured to update the plurality ofsecond images to obtain corresponding update features U.

An image de-superposing circuit 510 is connected with the superposedimage input terminal 503, the fourth convolutional neural networkcircuit 509 and the output terminal 511 and configured to performde-superposing on the superposed image A according to the updatefeatures U and their superposing weights to obtain the first image UL,and output the first image UL through the output terminal 511.

In an embodiment of the present disclosure, the image de-superposingcircuit 510 is configured to multiply the update features U by thesecond weight parameter b to obtain a second product bU, remove thesecond product from the superpoed image A to obtain a first product(A-bU), and divide the first product (A-bU) by the first weightparameter a to obtain the first image UL; where the first weightparameter is greater than 0, and a sum of the first weight parameter andthe second weight parameter is 1. That is, in the image superposingcircuit 510:

UL=(A−bU)/a  Expression 3

a+b=1 and a>0  Expression 4

That is to say, the image de-superposing circuit 510 and the imagesuperposing circuit 408 perform inverse processing, wherein the firstweight parameter and the second weight parameter satisfy a samecondition. Thus, the first image UL output by the image de-superposingcircuit 510 can be the same as the first image obtained and split fromthe original image.

A muxer 502 is connected with respective output terminals 511-514 andthe output interfaces 515 and configured to join the first image UL andthe plurality of second images UR, BR, BL to obtain a decoded image 501,and the decoded image 501 is output through the output interface 515.

As described above, the third convolutional neural network circuit 507and the fourth convolutional neural network circuit 509 in the imagedecoding apparatus 50 shown in FIG. 5 have the same filter parameter asthe second convolutional neural network circuit 409 and the firstconvolutional neural network circuit 407 in the image encoding apparatus40 shown in FIG. 4, and the de-superposing process performed by theimage de-superposing circuit 510 in the image decoding apparatus 50shown in FIG. 5 is completely inverse to the superposing processperformed by the image superposing circuit 408 in the image encodingapparatus 40 shown in FIG. 4, and the de-difference process performed bythe de-difference circuit 508 in the image decoding apparatus 50 shownin FIG. 5 is completely inverse to the difference acquisition processperformed by the image difference acquisition circuit 410 in the imageencoding apparatus 40 shown in FIG. 4, that is, an image encoded andcompressed by the image encoding apparatus 40 shown in FIG. 4 can bedecoded and recovered exactly by the image decoding apparatus 50 shownin FIG. 5, having nothing to do with the filter parameters of therespect convolutional neural networks.

In the first embodiment of the present disclosure, the updating processis completed by the first convolutional neural network circuit 407, andthe prediction process is completed by the second convolutional neuralnetwork circuit 409. In specific applications, by performingcorresponding training on the first convolutional neural network circuit407 and the second convolutional neural network circuit 409, the firstconvolutional neural network circuit 407 and the second convolutionalneural network circuit 409 have optimized filter parameters, so that theimage encoding apparatus has a higher compression rate, withoutartificially setting corresponding filter parameters, which reducescomplexity in setting the filter parameters.

In the first embodiment of the present disclosure, a weight parameter aused for image superposing is set, so as to further enhance downscalingand upscaling performance and flexibility of the encoder and thedecoder.

Additionally, in the first embodiment of the present disclosure, basedon appropriate training, the output difference features D_(n), D_(d) andD_(v) are approximately 0, and the overall compression rate can be closeto 75%. In splitting process performed by the demuxer 402 shown in FIG.4, the images UR, BR and BL are predicted by the first image UL at theupper left side. The present disclosure is not limited thereto. In analternative embodiment, the images UR, BR and BL can also be used as thefirst image to predict other images. In addition, in the firstembodiment of the present disclosure, the number of the first imageis 1. The present disclosure is not limited thereto. In an alternativeembodiment, two images can also be used to predict two images, or threeimages can be used to predict three images. Since the number of obtainedsuperposed images is the same as the number of the first images, in thecase of only one image compression circuit, if adopting two images topredict two images, the theoretical maximum value of the compressionrate is 50%, and if adopting three images to predict three images, thetheoretical maximum value of the compression rate is 25%.

In the first embodiment of the present disclosure, a single-stagecompression system composed of the first convolutional neural networkcircuit 407 and the second convolutional neural network circuit 409 isused. The present disclosure is not limited thereto. In an alternativeembodiment, two stages and more stages of compression configuration canbe used.

As described above, the image decoding apparatus 50 shown in FIG. 5accurately decodes and recovers the images encoded and compressed by theimage encoding apparatus 40 shown in FIG. 4, that is, the image encodingapparatus 40 shown in FIG. 4 and the image decoding apparatus 50 shownin FIG. 5 are compose a lossless system. In actual applications, in theStandard such as JPEG2000, it needs to perform quantization process onthe encoded data, and then approximate the quantized encoded data tobeing decoded, so as to form a lossy system on the whole.

FIG. 6 is a schematic diagram illustrating structure of an imageprocessing system according to the second embodiment of the presentdisclosure. An image processing system 6 according to the secondembodiment of the present disclosure shown in FIG. 6 comprises the imageencoding apparatus 40 shown in FIG. 4 and the image decoding apparatus50 shown in FIG. 5, and the image processing system 6 according to thesecond embodiment of the present disclosure further comprises aquantization apparatus 60.

As shown in FIG. 6, the quantization apparatus 60 is connected with theimage encoding apparatus 40 and the image decoding apparatus 50. Thestructure and inputs/outputs of the image encoding apparatus 40 in FIG.6 are the same as the description by referring to FIG. 4, and thus therepetitive parts are omitted.

The quantization apparatus 60 is connected with the image encodingapparatus 40 and configured to receive the superposed image A and thedifference features D_(n), D_(d) and D_(v) output from the outputinterface 411, perform quantization process and inverse quantizationprocess on the superposed image A and the difference features D_(n),D_(d) and D_(v), to generate quantization superposed images andquantization difference features.

In particular, the quantization apparatus is configured to utilize theuniform scalar quantitation USQ function to perform the quantizationprocess on the superposed image and the difference features,

$\begin{matrix}{{{USQ}(x)} = {{{sign}(x)}\left\lfloor \frac{x}{\delta} \right\rfloor}} & {{Expression}\mspace{14mu} 5}\end{matrix}$

where

${{sign}(x)} = \left\{ {\left. \begin{matrix}1 & {x > 0} \\0 & {x = 0} \\{- 1} & {x < 0}\end{matrix} \right|,\left\lfloor \frac{x}{\delta} \right\rfloor} \right.$

is the smallest integer smaller than x, and δ is a quantizationparameter.

The quantization process represented by the Expression 5 complies withthe JPEG 2000 standard.

According to the JPEG 2000 standard, the quantization apparatus isconfigured to utilize the inverse uniform scalar quantization InvUSQfunction to perform the inverse quantization process on the output q ofthe uniform scalar quantization USQ function to generate thequantization superposed images and the quantization difference features,where

InvUSQ(q)=sign(q)(|q|+0.5)δ  Expression 6

As described above, by performing the corresponding training on theconvolutional neural network circuits in the encoding apparatus and thedecoding apparatus, corresponding filter parameters can be set for therespective convolutional neural network circuits. For the imageprocessing system according to the second embodiment shown in FIG. 6,since the quantization parameter δ is used, it needs to provide atraining method which is capable of simultaneously training therespective convolutional neural network circuits and the quantizationcircuits in the encoding apparatus and the decoding apparatus.

FIG. 7 illustrates a flowchart diagram of a training method according toa third embodiment of the present disclosure. As shown in FIG. 7, thetraining method according to the third embodiment of the presentdisclosure comprises following steps.

In step S701, a fixed quantization parameter δ is selected. In the thirdembodiment of the present disclosure, a relatively large initial value(e.g. 1000) of the quantization parameter δ is selected, so that theoutput is similar to lossless system and the optimization problem willbe easy to solve. After that, the process moves to step S702.

In step S702, in the case of a fixed quantization parameter δ, thetraining images are input to the image processing system, weight valuesof respective filter circuits at respective convolutional layers in thefirst to fourth convolutional neural network circuits are adjusted, anda fixed number of iterations are run to optimize a target function.

The target function is as follows:

θ=arg_(θ)min_(X)MSE(X,OUT_(θ)(X,δ))  Expression 7

X represents input training images, OUT represents output images, andMSE is a mean square error function between the input training imagesand the output images. After that, the process moves to the step S703.

In step S703, the quantization parameter is reduced by a predeterminedvalue. For example, the quantization parameter can be reduced by 5%.After that, the process moves to step S704.

In step S704, it is determined whether the quantization parameter is notsmaller than the predetermined threshold. The predetermined threshold isdetermined in advance, e.g., 1.

If a positive result is obtained in step S704, i.e., the quantizationparameter is not smaller than a predetermined threshold, then theprocess returns to the step S702 to repeat the training in step S702.

On the contrary, if a negative result is obtained in step S704, i.e.,the quantization parameter is small enough, the training process endsup.

It can be seen from the training method according to the thirdembodiment of the present disclosure shown in FIG. 7, the training aimis to reduce both the MSE and the quantization parameter δ. If the MSEis 0, then the quantization parameter δ is very large; if thequantization parameter δ is very small, then the MSE is large.Therefore, in the training process, it needs to make an appropriatecompromise between the MSE and the quantization parameter δ based on anacceptable quantization compression level and image quality.

Additionally, the image encoding/decoding apparatus and the imageprocessing system as described above by referring to FIGS. 4-6 can befurther used for the display apparatus according to an embodiment of thepresent disclosure. The display apparatus according to an embodiment ofthe present disclosure can be any product or means having the displayfunction such as a mobile phone, a tablet computer, a television set, adisplay, etc.

The image encoding apparatus, the image decoding apparatus, the imageprocessing system comprising the image encoding/decoding apparatus, thetraining method for the image processing system and the displayapparatus are described above by referring to the accompanying figures.The image encoding apparatus, the image decoding apparatus, and theimage processing system comprising the image encoding/decoding apparatusaccording to the present disclosure enhance the downscaling andupscaling performance and flexibility of the encoder and the decoderbased on the new weight parameters, so as to further improve theperformance of the overall system. In addition, the training method forthe image processing system according to the present disclosure realizeshigher compression performance by optimizing the compression efficiencyof the overall system in the case of different quantization parameters.

In the above descriptions, the image encoding apparatus, the imagedecoding apparatus, the image processing system and the circuitsinvolved therein can be implemented by a central processor (CPU), orrespective apparatuses and circuits can be implemented bymicroprocessors such as a digital signal processor (DSP), a fieldprogrammable gate array (FPGA) or the like.

It needs to note that in the present specification, terms “comprise”,“include” or any other variant thereof intends to cover non-exclusivecontaining, such that a process, method, object or device comprising aseries of elements not only comprise those elements, but also compriseother elements not listed explicitly, or also comprise elements inherentto this process, method, object or device. Without more limitations, anelement defined by an expression of “comprise a/an . . . ” does notexclude that additional same element exists in the process, method,object and device comprising the element.

Finally, it needs to note that the above series of processes not onlycomprise processes performed in a time sequence according to the orderdescribed herein, but also comprise processes performed in parallel orseparately in a non-time sequence.

According to the description of the above implementations, it is clearfor those skilled in the art to known that the present disclosure can berealized by means of software together with necessary hardwareplatforms, or can be realized by only hardware. Based on suchunderstanding, all or part of the technical solutions of the presentdisclosure that contribute to the background art can be reflected in aform of a computer software product. The computer software product canbe stored in a storage media, such as ROM/RAM, magnetic disk, opticaldisk, etc., comprising several instructions, which are used to make onepiece of computer equipment (it may be a personal computer, a server, ora network device, etc.) execute the method as disclosed in respectiveembodiments or some parts of the embodiments of the present disclosure.

The above are detailed description of the present disclosure. Thespecification applies a specific example to describe principles andimplementations of the present disclosure. The description of the aboveembodiments is just used to help in understanding the principles andcore concepts of the present disclosure. At the same time, for thoseordinary skilled in the art, based on the concept of the presentdisclosure, change would occur to the specific implementations andapplication scopes. To sum up, the content of the present specificationshall not be understood as a limitation to the present disclosure.

1-26. (canceled)
 27. An image encoding method, comprising steps of:acquiring a first image and a plurality of second images; updatingfeatures of each second image of the plurality of second images toobtain corresponding update features; superposing the first image withthe update features of each second image of the plurality of secondimages to generate superposed images; generating a plurality ofprediction images according to the superposed images; determiningdifference features between each second image of the plurality of secondimages and a corresponding prediction image; outputting the superposedimages and the difference features; wherein the updating and/orpredicting adopts a convolutional neural network.
 28. The image encodingmethod according to claim 27, comprising a step of: splitting an inputoriginal image into the first image and the plurality of second images.29. An image decoding method, comprising steps of: receiving superposedimages and difference features; generating a plurality of predictionimages according to the superposed images; generating a plurality ofsecond images according to each prediction image of the plurality ofprediction images and the difference features; updating each secondimage of the plurality of second images to obtain corresponding updatefeatures; performing de-superposing on the superposed images accordingto the update features to obtain a first image; outputting the pluralityof second images and the first image; wherein the updating and/orpredicting adopts a convolutional neural network.
 30. The image decodingmethod according to claim 29, comprising a step of: joining the firstimage and the plurality of second images to obtain a decoded image. 31.An image processing system, comprising: an image encoding apparatus,comprising: a first image input terminal, configured to acquire a firstimage; a plurality of second image input terminals, configured toacquire a plurality of second images; a first feature circuit, connectedwith the plurality of second image input terminals and configured toupdate features of each second image of the plurality of second imagesto obtain corresponding update features; an image superposing circuit,connected with the first image input terminal and the firstconvolutional neural network circuit and configured to superpose theupdate feature of each second image of the plurality of second imageswith the first image to generate superposed images and output thesuperposed images; a first prediction circuit, connected with the imagesuperposing circuit and configured to generate a plurality of predictionimages according to each of the superposed images; an image differenceacquisition circuit, connected with the plurality of second image inputterminals and the prediction circuit and configured to determinedifference features between each second image of the plurality of secondimages and a corresponding prediction image and output the differencefeatures; an output interface, configured to output the superposedimages and the difference features; an image decoding apparatus,comprising: a superposed image input terminal, configured to receive thesuperposed images; a difference feature input terminal, configured toreceive the difference features; a second prediction circuit, connectedwith the superposed image input terminal and configured to generate aplurality of prediction images according to the superposed images; ade-difference circuit, connected with the difference feature inputterminal and the prediction circuit and configured to generate aplurality of fourth images according to each prediction image of theplurality of prediction images and the difference features, and outputthe plurality of fourth images; a second feature circuit, connected withthe de-difference circuit and configured to update the plurality offourth images to obtain corresponding update features; and an imagede-superposing circuit, connected with the superposed image inputterminal and the fourth convolutional neural network circuit andconfigured to perform de-superposing on the superposed images accordingto the update features to obtain a third image, and output the thirdimage; an output terminal, configured to output the plurality of fourthimages and the third image.
 32. The image processing system according toclaim 31, wherein the image encoding apparatus further comprises: ademuxer, connected with the first image input terminal and the pluralityof second image input terminals and configured to split an inputoriginal image to obtain the first image and the plurality of secondimages.
 33. The image processing system according to claim 32, whereinthe demuxer is configured to split an original image into 2n images, anumber of the first image is 1, a number of the second images is 2n−1,and n is an integer greater than
 0. 34. The image processing systemaccording to claim 31, wherein the image superposing circuit superposesthe update features of each second image of the plurality of secondimages with the first image according to a superposing weight.
 35. Theimage processing system according to claim 34, wherein the imagesuperposing circuit is configured to multiply the first image by a firstweight parameter to obtain a first product, multiply the update featuresby a second weight parameter to obtain a second product, and superposethe first product and the second product to generate an superposedimage; where the first weight parameter is greater than 0, and a sum ofthe first weight parameter and the second weight parameter is
 1. 36. Theimage processing system according to claim 31, wherein the imagedecoding apparatus further comprises: a muxer, connected with the outputterminal and configured to join the first image and the plurality ofsecond images to obtain a decoded image and output the decoded imagethrough an output interface.
 37. The image processing system accordingto claim 31, wherein the image de-superposing circuit is configured toperform de-superposing on the superposed images according to the updatefeatures and their superposing weights.
 38. The image processing systemaccording to claim 37, wherein the image de-superposing circuit isconfigured to multiply the update features by a second weight parameterto obtain a second product, remove the second product from thesuperposed images to obtain a first product, and divide the firstproduct by a first weight parameter to obtain the first image; where thefirst weight parameter is greater than 0, and a sum of the first weightparameter and the second weight parameter is
 1. 39. The image processingsystem according to claim 31, wherein the first prediction circuit is asecond convolutional neural network circuit, the second predictioncircuit is a third convolutional neural network circuit, the firstfeature circuit is a first convolutional neural network circuit, and thesecond feature circuit is a fourth convolutional neural network circuit.40. The image processing system according to claim 31, furthercomprising: a quantization apparatus, connected with the image encodingapparatus and configured to receive the superposed images and thedifference features output from the output interface, and performquantization process and inverse quantization process on the superposedimages and the difference features to generate quantization superposedimages and quantization difference features; and the image decodingapparatus, configured to output the quantization superposed image andthe quantization difference features to a superposed image inputterminal and a difference feature input terminal of the image decodingapparatus.
 41. The image processing system according to claim 31,wherein the quantization apparatus is configured to perform thequantization process on the superposed images and the differencefeatures by utilizing an uniform scalar quantization USQ function,${{USQ}(x)} = {{{sign}(x)}\left\lfloor \frac{x}{\delta} \right\rfloor}$where ${{sign}(x)} = \left\{ {\left. \begin{matrix}1 & {x > 0} \\0 & {x = 0} \\{- 1} & {x < 0}\end{matrix} \right|,\left\lfloor \frac{x}{\delta} \right\rfloor} \right.$is a smallest integer smaller than x, and δ is a quantization parameter.42. The image processing system according to claim 31, wherein thequantization apparatus is configured to perform the quantization processon an output q of the uniform scalar quantization USQ function byutilizing an inverse uniform scalar quantization InvUSQ function togenerate the quantization superposed images and the quantizationdifference features,where InvUSQ(q)=sign(q)(|q|+0.5)δ.
 43. A training method for the imageprocessing system according to claim 31, comprising: selecting a fixedquantization parameter; inputting a training image to the imageprocessing system, adjusting weight values of respective filter circuitsat respective convolutional layers in convolutional neural networkcircuits, and running a limited number of iterations to optimize atarget function; and reducing the quantization parameter by apredetermined value, and repeating a training step of optimizing thetarget function if the quantization parameter is not smaller than apredetermined threshold; otherwise, the training method ends up.
 44. Thetraining method according to claim 43, wherein the target function is:θ=arg_(θ)min_(X)MSE(X,OUT_(θ)(X,δ)) where X represents an input trainingimage, OUT represents an output image, and MSE is a mean square errorfunction between the input training image and the output image.
 45. Animage encoding apparatus performing the image encoding method accordingto claim
 27. 46. An image decoding apparatus performing the imagedecoding method according to claim 29.